Skip to content

[Test] In test_patching_cluster, reduce the risk of ICEs and facilitate troubleshooting - #7525

Merged
gmarciani merged 10 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3160/fix-tests-patching-0804-1
Aug 7, 2026
Merged

[Test] In test_patching_cluster, reduce the risk of ICEs and facilitate troubleshooting#7525
gmarciani merged 10 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3160/fix-tests-patching-0804-1

Conversation

@gmarciani

@gmarciani gmarciani commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Description of changes

In test_patching_cluster:

  1. reduce the risk of ICEs by using c5.xlarge for headnode and login nodes rather than GPU instances.
  2. increase the patching timeout for the head node to 3600s (from 1800s) to avoid the risk of timeouts when small instance types are used (such as c5.xlarge).
  3. print the full ImageBuilder log when the patching fails to facilitate troubleshooting.
  4. support new patching flavor minimal-capped and full-capped, where the patching is capped to the maximum kernel version supported by Lustre.
  5. in both develop and released test config, execute full-capped on both Ubuntu and RHEL/Rocky, and full (uncapped) on AL2023, where capping is not required.
  6. print stdout/stderr of patching script on the head node both in case of success and failure, to facilitate troubleshooting.
  7. Prevent false positives on Rocky8 and RHEL8 caused by lazily loaded kernel modules.

Notes

  1. skip-bad-url-suffix-check: safe to skip because the check would otherwise fail for the expected FSx Lsytre repo URL

Tests

SUCCESS

test-suites:
  patching:
    test_patching.py::test_patching_cluster:
      dimensions:
        - regions: [{{ g4dn_8xlarge_CAPACITY_RESERVATION_2_INSTANCES_2_HOURS_NOPG_ubuntu2404 }}]
          instances: ["g4dn.8xlarge"]
          oss: ["ubuntu2404"]
          flags: ["patching:minimal-capped"]
          schedulers: ["slurm"]
        - regions: [{{ g4dn_8xlarge_CAPACITY_RESERVATION_2_INSTANCES_2_HOURS_NOPG_ubuntu2404 }}]
          instances: ["g4dn.8xlarge"]
          oss: ["ubuntu2404"]
          flags: ["patching:full-capped"]
          schedulers: ["slurm"]
        - regions: [{{ g4dn_8xlarge_CAPACITY_RESERVATION_2_INSTANCES_2_HOURS_NOPG_rhel9 }}]
          instances: ["g4dn.8xlarge"]
          oss: ["rhel9"]
          flags: ["patching:full-capped"]
          schedulers: ["slurm"]
        - regions: [{{ g4dn_8xlarge_CAPACITY_RESERVATION_2_INSTANCES_2_HOURS_NOPG_rhel8 }}]
          instances: ["g4dn.8xlarge"]
          oss: ["rhel8"]
          flags: ["patching:full-capped"]
          schedulers: ["slurm"]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@gmarciani gmarciani added skip-changelog-update Disables the check that enforces changelog updates in PRs 3.x Test labels Aug 4, 2026
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3160/fix-tests-patching-0804-1 branch 2 times, most recently from 49cec12 to 033d622 Compare August 4, 2026 21:52
@gmarciani
gmarciani marked this pull request as ready for review August 4, 2026 21:55
@gmarciani
gmarciani requested review from a team as code owners August 4, 2026 21:55
@gmarciani gmarciani added the skip-bad-url-suffix-check Skip the checks regarding the bad URL suffix label Aug 4, 2026
…xlarge for hea dnode and login nodes rather than GPU instances.
… the patching fails to facilitate troubleshooting.
…e head node to 3600s (from 1800s)

to support slower patching when small instance type is used.
Patching the head node in place rebuilds the out-of-tree kernel modules
against the new kernel. A known issue in the EFA compilation makes that
rebuild memory hungry, so c5.xlarge is not enough.
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3160/fix-tests-patching-0804-1 branch from 033d622 to ad516c2 Compare August 6, 2026 16:57
@hanwen-cluster

Copy link
Copy Markdown
Contributor

reduce the risk of ICEs by using c5.xlarge for headnode and login nodes rather than GPU instances.

Minor: since this commit reduces the number of tests running in parallel, ICE won't be a problem. In the past, ICE happened because we made capacity reservation for 3 instances only, while we needed many more

hanwen-cluster
hanwen-cluster previously approved these changes Aug 6, 2026
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3160/fix-tests-patching-0804-1 branch from 43341b4 to d2b897c Compare August 6, 2026 18:49
hanwen-cluster
hanwen-cluster previously approved these changes Aug 6, 2026
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3160/fix-tests-patching-0804-1 branch from d2b897c to ad516c2 Compare August 6, 2026 21:39
…od-probing kernel modules that are lazily loaded.
@gmarciani
gmarciani enabled auto-merge (rebase) August 7, 2026 14:18
@gmarciani
gmarciani merged commit a464f7b into aws:develop Aug 7, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.x skip-bad-url-suffix-check Skip the checks regarding the bad URL suffix skip-changelog-update Disables the check that enforces changelog updates in PRs Test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants